(IcebergIO) Support PartitionSpec/SortOrder on dynamic table creation via IcebergIO - #39408
Conversation
ahmedabu98
left a comment
ahmedabu98
left a comment
There was a problem hiding this comment.
Sorry was OOO for some time and just seeing this now. I think this PR is a good addition, and doesn’t harm the portable case at all. Just had one nit regarding naming but it LGTM! Feel free to convert it so we can run some tests against it
Also I’m curious to know if the ManagedIO route is causing any friction for your use case?
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
sounds good! Just renamed the method and undrafted the PR. No, so far so good! The iceberg write API in Scio goes through ManagedIO and works well. I've just been parity testing a lot of iceberg IO features on both managedIO + icebergIO and caught this particular discrepancy! |
2 participants
#38269 added SortOrder support to Iceberg writes via Managed IO, but it's not accessible through IcebergIO directly. This PR adds support for setting SortOrder+PartitionSpec through the IcebergIO.writeRows builder.
It does make serialization of the transform more complex which might be antithetical to the goal of keeping new transforms as portable as possible -- cc @ahmedabu98, let me know if not exposing these params via
IcebergIOwas by design and if I should just switch to ManagedIO if I need to specify.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.